Image processing circuit and image processing method

ABSTRACT

The present disclosure provides an image processing circuit including a neural network processor, a background processing circuit and a blending circuit. The neural network processor is configured to process input image data to determine whether the input image data has a predetermined object so as to generate to heat map. The background processing circuit blurs the input image data to generate blurred image data. The blending circuit blends the input image data and the blurred image data according to the heat map to generate output image data.

This application claims the benefit of U.S. Provisional Ser. No.63/242,471, filed on Sep. 9, 2021, the subject matter of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present disclosure relates to an image processing circuit and animage processing method that partially blur a background of image data.

Description of the Related Art

Among current electronic apparatuses having an image capturing function,some electronic apparatuses have a background blur function so as topartially blur the background of a captured image. For example, during avideo conference, an electronic apparatus may initiatively analyze imagedata captured by a camera so as to identify the location of anindividual for such background blur to provide user privacy protection.However, the background blur may result display flaws at foregroundedges due to foreground identification issues, such that video qualitymay be degraded.

SUMMARY OF THE INVENTION

Therefore, it is an object of the present disclosure to provide an imageprocessing method, which generates a heat map by a neural networkprocessor and then blends an original image and a blurred imageaccording to the heat map so as to generate a background blurred image.

An image processing circuit disclosed according to one embodiment of thepresent disclosure includes a neural network processor, a backgroundprocessing circuit and a blending circuit. The neural network processorprocesses input image data to determine whether the input image data hasa predetermined object so as to generate to heat map. The backgroundprocessing circuit blurs the input image data to generate blurred imagedata. The blending circuit blends the input image data and the blurredimage data according to the heat map to generate output image data.

An image processing method disclosed according to one embodiment of thepresent disclosure includes processing input image data by a neuralnetwork processor to determine whether the input image data has apredetermined object so as to generate to heat map, blurring the inputimage data to generate blurred image data, and blending the input imagedata and the blurred image data according to the heat map to generateoutput image data.

With the foregoing embodiments of the present disclosure, using preciseidentification capabilities of a neural network processor and abackground processing circuit having a simple structure, a backgroundcan be quickly and effectively blurred while clarity of a foreground ismaintained, and foreground edges of the background blurred image canalso appear smoother.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an image processing circuit accordingto an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a neural network processor according toan embodiment of the present disclosure;

FIG. 3 is a schematic diagram of operations of a convolutional neuralnetwork circuit and a rear-end processing circuit in a neural networkprocessor according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a background processing circuitaccording to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of input image data, a scaled heat map,scaled blurred image data and output image data according to anembodiment of the present disclosure; and

FIG. 6 is a flowchart of an image processing method according to anembodiment of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a schematic diagram of an image processing circuit 100according to an embodiment of the present disclosure. As shown in FIG. 1, the image processing circuit 100 includes a scaling circuit 110, aneural network processor 120, a background processing circuit 130,scaling circuits 140 and 150, and a blending circuit 160. In thisembodiment, the image processing circuit 100 is suitable for any imagecapturing device or image display device such as a camera, a smartphone, a tablet computer and a laptop computer. That is, the imageprocessing circuit 100 can receive input image data Din from an imagesensor or generated by an image source so as to generate output imagedata Dout, which may be displayed on a display panel.

In an operation of the image processing circuit 100, the scaling circuit110 downscales the input image data Din to generate downscaled inputimage data Din′. For example, assuming that the input image data Dinincludes frames having a resolution of 1920*1080 and the scaling circuit110 can reduce the resolution of the input image data Din, thedownscaled input image data Din′ then includes frames having a lowerresolution so as to decrease the amount of data to be processedsubsequently. Then, the neural network processor 120 performs imageidentification on the downscaled input image data Din′ to determinewhether each frame in the downscaled input image data Din′ has apredetermined object so as to generate a heat map. In this embodiment,the predetermined object is a person, and the following description isgiven on this basis.

In one embodiment, as shown in FIG. 2 , the neural network processor 120includes a format conversion circuit 210, a convolutional neural network(CNN) circuit 220 and a rear-end processing circuit 220. In an operationof the neural network processor 120, the format conversion circuit 210performs format conversion on the downscaled input image data Din′ tomeet a requirement of the neural network processor 120, for example,converting an original NV12 format to an RGB format. The convolutionalneural network circuit 220 performs feature extraction andidentification on the downscaled input image data Din′ to identify theposition(s) and edges of one or more persons in the downscaled inputimage data Din′ and to accordingly generate one or more feature maps. Inone embodiment, the convolutional neural network circuit 220 generates aperson feature map and an edge feature map. The rear-end processingcircuit 230 fuses and converts the one or more feature maps generated bythe convolutional neural network circuit 220 to generate a heat map DH,which indicates an area including one or more persons in the downscaledinput image data Din′. For example, the heat map DH and the downscaledinput image data Din′ may have the same resolution; an area containingone or more persons in the downscaled input image data has a highervalue (for example 255) in the corresponding heat map DH, and theremaining area in the downscaled input image data Din′ has a lower value(for example, 0) in the corresponding heat map DH.

In one embodiment, refer to FIG. 3 showing operations of theconvolutional neural network circuit 220 and the rear-end processingcircuit 230. As shown in FIG. 3 , the downscaled input image data Din′may first undergo operations of different convolution layerscorresponding to multiple partial residual blocks 312_1 to 312_N in abackbone 310 to generate corresponding feature values, and an atrousspatial pyramid pooling (ASPP) 320 then receives the feature valuegenerated from the backbone 310 so as to reduce dimensions of thefeature maps and reinforce the feature values. Next, a semanticestimation module 330 performs multiple feature fusions 332_1 to 332_Mon the feature values from the backbone 310 and the ASPP 320 to generatea person feature map. A border correction module 340 performs edgeidentification on the person feature map to generate an edge featuremap. Lastly, a feature value capturing module 352 in a heat mapestimation module 350 captures an output feature map according to theperson feature map and/or the edge feature map, and a heat mapgenerating module 354 then generates the heat map DH according to theoutput feature map. In one embodiment, the backbone 310, the ASPP 320,the semantic estimation module 330 and the border correction module 340may be implemented by the convolutional neural network circuit 220 or beincluded in the convolutional neural network circuit 220, and the heatmap estimation map 350 may be implemented by the rear-end processingcircuit 230 or be included in the rear-end processing circuit 230. Inone embodiment, the rear-end processing circuit 230 is also implementedby a convolutional neural network circuit, or may be integrated in theconvolutional neural network circuit 220.

Referring to FIG. 3 showing the schematic diagram of operations, theprocess of generating the heat map DH by the neural network processor120 may be further divided into three parts—first generating the personfeature map according to the downscaled input image data Din′, thengenerating the edge feature map according to the person feature map, andlastly generating the heat map DH according to the person feature mapand the edge feature map. By dividing and the process of generating theheat map DH into three parts processing accordingly, the accuracy forperson and edge identification can be improved and the processingcomplexity of the neural network processor 120 can be reduced. Inpractice, the neural network processor 120 may performing training withrespect to the three parts above; that is, providing a reference personfeature map for the neural network processor 120 to learn and identifypersons and generate the person feature map, providing a reference edgefeature map for the neural network processor 120 to learn and identifyedges from the person feature map so as to generate the edge featuremap, and providing a reference heat map for the neural network processor120 to learn and use the person feature map and the edge feature map togenerate the heat map DH.

The background processing circuit 130 blurs the downscaled input imagedata Din′ to generate blurred image data DBB. In one embodiment, thebackground processing circuit 130 may perform low-pass filtering on eachpixel in the downscaled input image data Din′, that is, performingweighted addition on each pixel and multiple surrounding pixels toobtain a filtered pixel value of the pixel. In one embodiment, thelow-pass filtering above may be performed via a Gaussian filter matrix.In another embodiment, as shown in FIG. 4 , the background processingcircuit 130 may include two scaling circuits 410 and 420. The scalingcircuit 410 may downscale the downscaled input image data Din′, and thescaling circuit 420 may perform upscaling to generate the blurred imagedata DBB. In one embodiment, the scaling circuits 410 and 420 mayoperate for multiple times to generate the blurred image data DBB, thatis, the blurred image data DBB may be again input to the scaling circuit410 for scaling of a next round to generate a next set of blurred imagedata DBB for output of the background processing circuit 130. In thisembodiment, the background processing circuit 130 directly blurs theentire of the downscaled input image data Din′ instead of selectivelyblurring a partial area of the downscaled input image data Din′. Morespecifically, for each frame in the downscaled input image data Din′,the background processing circuit 130 directly blurs the entire frame.Since the same processing is performed on the entire frame, thebackground processing circuit 130 may be implemented by a simplestructure, hence achieving the image blur function without involving acomplex circuit structure and/or design.

Next, the scaling circuit 140 upscales the heat map DH to generate ascaled heat map DH′, wherein a resolution of the heat map DH′ is greaterthan that of the heat map DH. In one embodiment, the resolution of thescaled heat map DH′ is equal to the resolution of the input image dataDin; for example, the resolutions of the scaled heat map DH′ and theresolution of the input image data Din are both 1920*1080. Similarly,the scaling circuit 150 upscales the blurred image data DBB to generatescaled blurred image data DBB′. In one embodiment, the resolution of thescaled blurred image data DBB′ is equal to the resolution of the inputimage data Din; for example, the resolutions of the scaled blurred imagedata DBB′ and the resolution of the input image data Din are both1920*1080.

In the operation of the blending circuit 160, the blending circuit 160blends the input image data Din and the scaled blurred image data DBB′according to the scaled heat map DH′, for example, performing weightedaddition, to generate output image data Dout, wherein the scaled heatmap DH′ serves as a weight basis for the blending. For example, becausethe input image data Din, the scaled heat map DH′ and the scaled blurredimage data DBB′ have the same resolution, for pixels at the sameposition, a pixel value Pout of a pixel of the output image data Doutmay be calculated as below:

Pout=Pin*(PH/255)+PBB*((255−PH)/255)  (1)

where Pin is the pixel value of the input image data Din, PH is thepixel value of the scaled heat map DH′, and PBB is the pixel value ofthe scaled blurred image data DBB′. In an example, assuming that thepixel currently being processed is within an area of a person, PH isthen a very high value, for example, PH is “255”, and so the pixel valuePout of the output image data Dout calculated according to equation (1)above is the pixel value Pin of the input image data Din. In anotherexample, assuming that the pixel currently being processed is outside anarea of a person (for example, an area of the background), PH is then avery low value, for example, PH is “0”, and so the pixel value Pout ofthe output image data Dout calculated according to equation (1) above isthe pixel value PBB of the blurred image data DBB′.

In one embodiment, the blending performed by the blending circuit 160 onthe input image data Din and the scaled blurred image data DBB′ selectsand outputs one between the pixel values of the respective correspondingpixels of the input image data Din and the scaled blurred image dataDBB′ in a pixel-by-pixel manner according to the scaled heat map DH′, asthe pixel value of the corresponding pixel of the output image dataDout. For example, the pixel value of the scaled heat map DH′ may be 255or 0. When the pixel value of a pixel is “255”, it means that the pixelis located within an area of a person; when the pixel value of a pixelis “0”, it means that the pixel is not located within an area of aperson. In this embodiment, when the pixel value of the pixel currentlybeing processed in the scaled heat map DH is “255”, the blending circuit160 selects and outputs the pixel value of the corresponding pixel inthe input data image Din as the pixel value of the corresponding pixelin the output image data Dout; when the pixel value of the pixelcurrently being processed in the scaled heat map DH′ is “0”, theblending circuit 160 selects and outputs the pixel value of thecorresponding pixel in the scaled blurred image data DBB′ as the pixelvalue of the corresponding pixel in the output image data Dout.

FIG. 5 shows a schematic diagram of the input image data Din, the scaledheat map DH′, the scaled blurred image data DBB′ and the output imagedata Dout. As shown in FIG. 5 , with the processing of the imageprocessing circuit 100, the background can be quickly ad effectivelyblurred while the clarity of the foreground (person) is maintained.

It should be noted that, in the embodiments in FIG. 1 to FIG. 5 , theinput image data Din is first processed by the scaling circuit 110 togenerate the downscaled input imaged data Din′ that then enters theneural network processor 120 and the background processing circuit 130for subsequent processing, so as to reduce the overall circuitcomputation amount. However, this feature is not construed to be alimitation of the present disclosure. In other embodiments, the neuralnetwork processor 120 and the background processing circuit 130 maydirectly process the input image data Din. At this point, the scalingcircuits 110, 140 and 150 may be eliminated from the image processingcircuit 100, and the blending circuit 160 then directly blends the inputimage data Din and the blurred image data DBB according to the heat mapDH to generate the output image data Dout. In one embodiment, providedwith a circuit processing speed that is fast enough, the scaling circuit140 and the scaling circuit 150 may be implemented by the same circuit.

FIG. 6 shows a flowchart of an image processing method according to anembodiment of the present disclosure. The image processing method of thepresent disclosure is applicable to an image processing device. Withreference to the details of the embodiments above, the process of theimage processing method is as below.

In step 600, the process begins.

In step 602, input image data is processed by a neural network processorto determine whether the input image data has a predetermined object soas to generate to a heat map.

In step 604, the input image data is blurred to generate blurred imagedata.

In step 606, the input image data and the blurred image data are blendedaccording to the heat map to generate output image data.

Summarizing the present disclosure, in the image processing circuit andthe image processing method of the present disclosure, an original imageis analyzed by a neural network processor to generate a heat map, theoriginal image is blurred to generate a blurred image, and the originalimage and the blurred image are blended according to the heat map togenerate a background blurred image as output image data. In the presentdisclosure, using precise identification capabilities of a neuralnetwork processor and a background processing circuit having a simplestructure, a background can be quickly and effectively blurred whileclarity of a foreground is maintained without involving an additionalsensor (for example, a distance sensor) or selectively blurring an imageframe by a complicated circuit, and foreground edges of the backgroundblurred image can also appear smoother.

The description above provides merely preferred embodiments of thepresent disclosure, and all variations and modifications made based onthe range of claims of the present invention are to be encompassedwithin the scope of the present disclosure.

What is claimed is:
 1. An image processing circuit, comprising: a neural network processor, configured to process input image data to determine whether the input image data has a predetermined object so as to generate to heat map; a background processing circuit, configured to blur the input image data to generate blurred image data; and a blending circuit, configured to blend the input image data and the blurred image data according to the heat map to generate output image data.
 2. The image processing circuit of claim 1, wherein the background processing circuit blurs an entire of the input image data to generate the blurred image data.
 3. The image processing circuit of claim 1, wherein the neural network processor comprises: a convolutional neural network circuit, configured to perform feature extraction and identification on the input image data to identify a position of the predetermined object so as to generate an object feature map, and performing edge identification on the object feature map to generate an edge feature map; and a rear-end processing circuit, configured to generate the heat map according to the object feature map and the edge feature map.
 4. The image processing circuit of claim 1, wherein the neural network processor performs object identification on the input image data to generate an object feature map, performs edge identification on the object feature map to generate an edge feature map, and generates the heat map according to the object feature map and the edge feature map.
 5. The image processing circuit of claim 1, wherein the background processing circuit performs low-pass filtering on each pixel in the input image data to generate the blurred image data.
 6. The image processing circuit of claim 1, wherein the background processing circuit downscales and upscales the input image data for at least one round to generate the blurred image data.
 7. The image processing circuit of claim 1, wherein the blending circuit performs weighted addition on the input image data and the blurred image data to generate the output image data, wherein a weight used in the weighted addition is generated according to the heat map.
 8. The image processing circuit of claim 1, wherein the blending circuit selects and outputs one between pixel values of respective corresponding pixels of the input image data and the blurred image data in a pixel-by-pixel manner according to the heat map, as a pixel value of a corresponding pixel of the output image data.
 9. The image processing circuit of claim 1, further comprising: a first scaling circuit, configured to downscale the input image data to generate downscaled input image data; wherein the neural network processor processes the downscaled input image data to determine whether the downscaled input image data has the predetermined object so as to generate the heat map, and the background processing circuit blurs the downscaled input image data to generate the blurred image data.
 10. The image processing circuit of claim 9, further comprising: a second scaling circuit, configured to upscale the heat map to generate a scaled heat map; and a third scaling circuit, configured to upscale the blurred image data to generate scaled blurred image data; wherein, the blending circuit blends the input image data and the scaled blurred image data according to the heat map to generate the output image data.
 11. An image processing method, applied to an image processing device, the method comprising: processing input image data by a neural network processor to determine whether the input image data has a predetermined object so as to generate to heat map; blurring the input image data to generate blurred image data; and blending the input image data and the blurred image data according to the heat map to generate output image data.
 12. The image processing method of claim 11, wherein the blurring of the input image data blurs an entire of the input image data to generate the blurred image data.
 13. The image processing method of claim 11, wherein the neural network processor performs object identification on the input image data to generate an object feature map, performs edge identification on the object feature map to generate an edge feature map, and generates the heat map according to the object feature map and the edge feature map.
 14. The image processing method of claim 11, wherein the blending of the input image data and the blurred image data according to the heat map performs weighted addition on the input image data and the blurred image data to generate the output image data, wherein a weight used in the weighted addition is generated according to the heat map.
 15. The image processing method of claim 11, wherein the blending of the input image data and the blurred image data according to the heat map selects and outputs one between pixel values of respective corresponding pixels of the input image data and the blurred image data in a pixel-by-pixel manner according to the heat map, as a pixel value of a corresponding pixel of the output image data. 